De-congesting data centers with wireless point-to-multipoint flyways

ABSTRACT

In one embodiment, a source top-of-rack (ToR) switch may identify multiple destination ToR switches from a group of ToR switches to send data traffic to. The source ToR switch may be connected to the group of ToR switches via a base network. The system may determine whether each destination ToR switch is suitable for receiving data transmission via a point-to-multipoint wireless flyway. The two or more destination ToR switches that are determined to be suitable may be considered flyway candidate ToR switches. The system may establish the point-to-multipoint wireless flyway between the source ToR switch and the flyway candidate ToR switches. The system may then transmit the data traffic from the source ToR switch to each of the flyway candidate ToR switches via the point-to-multipoint wireless flyway.

TECHNICAL FIELD

The present disclosure relates generally to data centers, and morespecifically relates to exchanging data traffic among servers.

BACKGROUND

One of the challenges in building large data centers is that the cost ofproviding the same communication bandwidth between an arbitrary pair ofservers grows in proportion to the size of the cluster. In top-of-rack(ToR) architecture used for data centers, the ToR switch that connects agroup of servers in a given server rack to other ToR switches andservers may need to exchange data with other ToR switches. However,increasing the utilization of ToRs while preventing debilitating networkcongestion can be difficult. Since ToRs typically exchange large datavolumes with only a few other ToRs in the data center at any given pointin time, the sparse nature of the demand matrix may translate tosubstantial bottlenecks under a conventional data center topology. Inother words, as a handful of ToRs lag behind, they can hold back theentire network from completing its tasks.

This implies that it can be costly to scale data centers to accommodateemerging distributed computing applications. As the number of servers,server racks, and other pieces of equipment increases, it becomes morecostly and unmanageable to run the data center at a similar performancelevel.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe the manner in which the above-recited and otheradvantages and features of the disclosure can be obtained, a moreparticular description of the principles briefly described above will berendered by reference to specific embodiments thereof which areillustrated in the appended drawings. Understanding that these drawingsdepict only exemplary embodiments of the disclosure and are nottherefore to be considered to be limiting of its scope, the principlesherein are described and explained with additional specificity anddetail through the use of the accompanying drawings in which:

FIG. 1 illustrates an example network device according to some aspectsof the subject technology;

FIGS. 2A-B illustrate example system embodiments according to someaspects of the subject technology;

FIG. 3 illustrates a schematic block diagram of an example networkarchitecture for a data center;

FIG. 4 illustrates a schematic diagram of an example MU-MIMO network;

FIG. 5 illustrates an example server rack coupled with a wirelesstransceiver;

FIG. 6 illustrates a schematic block diagram of an example configurationfor a data center augmented by wireless flyways;

FIG. 7 illustrates a schematic diagram of an example data centerperforming a DL-MU-MIMO transmission among the ToR switches;

FIG. 8 illustrates a schematic diagram of an example data centerperforming a UL-MU-MIMO transmission among the ToR switches;

FIG. 9 illustrates an example method embodiment of performing aDL-MU-MIMO transmission;

FIG. 10 illustrates an example method embodiment of performing aUL-MU-MIMO transmission; and

FIG. 11 illustrates an example method embodiment of routing reversetraffic from a destination ToR switch to a source ToR switch.

DESCRIPTION OF EXAMPLE EMBODIMENTS

Various embodiments of the disclosure are discussed in detail below.While specific implementations are discussed, it should be understoodthat this is done for illustration purposes only. A person skilled inthe relevant art will recognize that other components and configurationsmay be used without parting from the spirit and scope of the disclosure.

Overview

In some embodiments, a source top-of-rack switch (ToR) may identifymultiple destination ToR switches from a group of ToR switches to senddata traffic to. The source ToR switch may be connected to the group ofToR switches via a base network. The system may determine whether eachdestination ToR switch is suitable for receiving data transmission via apoint-to-multipoint wireless flyway. The two or more destination ToRswitches that are determined to be suitable may be considered flywaycandidate ToR switches. The system may establish the point-to-multipointwireless flyway between the source ToR switch and the flyway candidateToR switches. The system may transmit the data traffic from the sourceToR switch to each of the flyway candidate ToR switches via thepoint-to-multipoint wireless flyway.

In some other embodiments, the system may identify multiple source ToRswitches, each of them having respective data to transmit to adestination ToR switch. The source ToR switches and the destination ToRswitch may belong to a group of ToR switches that are interconnected toeach other via a wired base network. Based on respective channel stateinformation (CSI) received from each of the source ToR switches, thesystem may select two or more flyway candidate ToR switches from thesource ToR switches. The flyway candidate ToR switches may have abeam-forming coefficient that satisfies a threshold value. The systemmay establish the multipoint-to-point wireless flyway between the flywaycandidate ToR switches and the destination ToR switch. The destinationToR switch may then receive respective data symbols from the flywaycandidate ToR switches via the multipoint-to-point wireless flyway.

Description

A computer network is a geographically distributed collection of nodesinterconnected by communication links and segments for transporting databetween endpoints, such as personal computers and workstations. Manytypes of networks are available, with the types ranging from local areanetworks (LANs) and wide area networks (WANs) to overlay andsoftware-defined networks, such as virtual extensible local areanetworks (VXLANs).

LANs typically connect nodes over dedicated private communications linkslocated in the same general physical location, such as a building orcampus. WANs, on the other hand, typically connect geographicallydispersed nodes over long-distance communications links, such as commoncarrier telephone lines, optical lightpaths, synchronous opticalnetworks (SONET), or synchronous digital hierarchy (SDH) links. LANs andWANs can include layer 2 (L2) and/or layer 3 (L3) networks and devices.

The Internet is an example of a WAN that connects disparate networksthroughout the world, providing global communication between nodes onvarious networks. The nodes typically communicate over the network byexchanging discrete frames or packets of data according to predefinedprotocols, such as the Transmission Control Protocol/Internet Protocol(TCP/IP). In this context, a protocol can refer to a set of rulesdefining how the nodes interact with each other. Computer networks maybe further interconnected by an intermediate network node, such as arouter, to extend the effective “size” of each network.

The disclosed technology addresses the need in the art for managing datacenters. Disclosed are systems, methods, and computer-readable storagemedia for de-congesting data centers with wireless flyways, specificallypoint-to-multipoint flyways and/or multipoint-to-point flyways. A briefintroductory description of exemplary systems and networks, asillustrated in FIGS. 1, 2A, and 2B, is disclosed herein. A detaileddescription of point-to-multipoint and multipoint-to-point wirelessflyways, related concepts, and exemplary variations, will then follow.These variations shall be described herein as the various embodimentsare set forth. The disclosure now turns to FIG. 1.

FIG. 1 illustrates example network device 110 suitable for implementingthe present invention. Network device 110 includes master centralprocessing unit (CPU) 162, interfaces 168, and bus 115 (e.g., a PCIbus). When acting under the control of appropriate software or firmware,CPU 162 is responsible for executing packet management, error detection,and/or routing functions, such as mis-cabling detection functions, forexample. CPU 162 preferably accomplishes all these functions under thecontrol of software including an operating system and any appropriateapplications software. CPU 162 may include one or more processors 163such as a processor from the Motorola family of microprocessors or theMIPS family of microprocessors. In an alternative embodiment, processor163 is specially designed hardware for controlling the operations ofrouter 110. In a specific embodiment, memory 161 (such as non-volatileRAM and/or ROM) also forms part of CPU 162. However, there are manydifferent ways in which memory could be coupled to the system.

Interfaces 168 are typically provided as interface cards or networkinterface controllers (NICs). Generally, they control the sending andreceiving of data packets over the network and sometimes support otherperipherals used with the router 110. Among the interfaces that may beprovided are Ethernet interfaces, frame relay interfaces, cableinterfaces, DSL interfaces, token ring interfaces, and the like. Inaddition, various very high-speed interfaces may be provided such asfast token ring interfaces, wireless interfaces, Ethernet interfaces,Gigabit Ethernet interfaces, ATM interfaces, HSSI interfaces, POSinterfaces, FDDI interfaces and the like. Generally, these interfacesmay include ports appropriate for communication with the appropriatemedia. In some cases, they may also include an independent processorand, in some instances, volatile RAM. The independent processors maycontrol such communications intensive tasks as packet switching, mediacontrol and management. By providing separate processors for thecommunications intensive tasks, these interfaces allow mastermicroprocessor 162 to efficiently perform routing computations, networkdiagnostics, security functions, etc.

Although the system shown in FIG. 1 is one specific network device ofthe present invention, it is by no means the only network devicearchitecture on which the present invention can be implemented. Forexample, an architecture having a single processor that handlescommunications as well as routing computations, etc. is often used.Further, other types of interfaces and media could also be used with therouter.

Regardless of the network device's configuration, it may employ one ormore memories or memory modules (including memory 161) configured tostore program instructions for the general-purpose network operationsand mechanisms for roaming, route optimization and routing functionsdescribed herein. The program instructions may control the operation ofan operating system and/or one or more applications, for example. Thememory or memories may also be configured to store tables such asmobility binding, registration, and association tables, etc.

FIG. 2A and FIG. 2B illustrate example system embodiments. The moreappropriate embodiment will be apparent to those of ordinary skill inthe art when practicing the present technology. Persons of ordinaryskill in the art will also readily appreciate that other systemembodiments are possible.

FIG. 2A illustrates a conventional system bus computing systemarchitecture 200 wherein the components of the system are in electricalcommunication with each other using a bus 205. Example system 200includes a processing unit (CPU or processor) 210 and a system bus 205that couples various system components including the system memory 215,such as read only memory (ROM) 220 and random access memory (RAM) 225,to the processor 210. The system 200 can include a cache of high-speedmemory connected directly with, in close proximity to, or integrated aspart of the processor 210. The system 200 can copy data from the memory215 and/or the storage device 230 to the cache 212 for quick access bythe processor 210. In this way, the cache can provide a performanceboost that avoids processor 210 delays while waiting for data. These andother modules can control or be configured to control the processor 210to perform various actions. Other system memory 215 may be available foruse as well. The memory 215 can include multiple different types ofmemory with different performance characteristics. The processor 210 caninclude any general purpose processor and a hardware module or softwaremodule, such as module 1 (232), module 2 (234), and module 3 (236)stored in storage device 230, configured to control the processor 210 aswell as a special-purpose processor where software instructions areincorporated into the actual processor design. The processor 210 mayessentially be a completely self-contained computing system, containingmultiple cores or processors, a bus, memory controller, cache, etc. Amulti-core processor may be symmetric or asymmetric.

To enable user interaction with the computing device 200, an inputdevice 245 can represent any number of input mechanisms, such as amicrophone for speech, a touch-sensitive screen for gesture or graphicalinput, keyboard, mouse, motion input, speech and so forth. An outputdevice 235 can also be one or more of a number of output mechanismsknown to those of skill in the art. In some instances, multimodalsystems can enable a user to provide multiple types of input tocommunicate with the computing device 200. The communications interface240 can generally govern and manage the user input and system output.There is no restriction on operating on any particular hardwarearrangement and therefore the basic features here may easily besubstituted for improved hardware or firmware arrangements as they aredeveloped.

Storage device 230 is a non-volatile memory and can be a hard disk orother types of computer readable media which can store data that areaccessible by a computer, such as magnetic cassettes, flash memorycards, solid state memory devices, digital versatile disks, cartridges,random access memories (RAMs) 225, read only memory (ROM) 220, andhybrids thereof.

The storage device 230 can include software modules 232, 234, 236 forcontrolling the processor 210. Other hardware or software modules arecontemplated. The storage device 230 can be connected to the system bus205. In one aspect, a hardware module that performs a particularfunction can include the software component stored in acomputer-readable medium in connection with the necessary hardwarecomponents, such as the processor 210, bus 205, display 235, and soforth, to carry out the function.

FIG. 2B illustrates a computer system 250 having a chipset architecturethat can be used in executing the described method and generating anddisplaying a graphical user interface (GUI). Computer system 250 is anexample of computer hardware, software, and firmware that can be used toimplement the disclosed technology. System 250 can include a processor255, representative of any number of physically and/or logicallydistinct resources capable of executing software, firmware, and hardwareconfigured to perform identified computations. Processor 255 cancommunicate with a chipset 260 that can control input to and output fromprocessor 255. In this example, chipset 260 outputs information tooutput 265, such as a display, and can read and write information tostorage device 270, which can include magnetic media, and solid statemedia, for example. Chipset 260 can also read data from and write datato RAM 275. A bridge 280 for interfacing with a variety of userinterface components 285 can be provided for interfacing with chipset260. Such user interface components 285 can include a keyboard, amicrophone, touch detection and processing circuitry, a pointing device,such as a mouse, and so on. In general, inputs to system 250 can comefrom any of a variety of sources, machine generated and/or humangenerated.

Chipset 260 can also interface with one or more communication interfaces290 that can have different physical interfaces. Such communicationinterfaces can include interfaces for wired and wireless local areanetworks, for broadband wireless networks, as well as personal areanetworks. Some applications of the methods for generating, displaying,and using the GUI disclosed herein can include receiving ordereddatasets over the physical interface or be generated by the machineitself by processor 255 analyzing data stored in storage 270 or 275.Further, the machine can receive inputs from a user via user interfacecomponents 285 and execute appropriate functions, such as browsingfunctions by interpreting these inputs using processor 255.

It can be appreciated that example systems 200 and 250 can have morethan one processor 210 or be part of a group or cluster of computingdevices networked together to provide greater processing capability.

FIG. 3 illustrates a schematic block diagram of an example networkarchitecture for a data center. Although example architecture 300 willbe described as a top-of-rack (ToR) architecture, the disclosedembodiments can be practiced just as readily with other types of datacenter architecture such as end-of-row (EoR) or middle-of-row (MoR)topologies. Racks 302A, 302B, 302C (collectively “302”) may housemultiple types and instances of computing device modules such asswitches, servers, storage equipment, etc. For example, rack 302A mayconsist of multiple servers 308A-1, 308A-2, 308A-3, . . . , 308A-nconnected to top-of-rack (ToR) switch 306A via relative short copperand/or fiber-optic cabling. Similarly, rack 302B may have servers308B-1, 308B-2, 308B-3, . . . , 308B-n connected to ToR 306B while rack302C may house servers 308C-1, 308C-2, 308C-3, . . . , 308C-n and ToR306C.

One of ordinary skill in the art will understand that a ToR (i.e., ToRswitch) does not necessarily need to be physically located at the “top”of a rack, but it can be placed anywhere inside or outside the rack aslong as it has direct connectivity to other modules in the rack. One ofskill in the art will also understand that, although FIG. 3 depictsthree server racks 302, a data center may have any number of serverracks or cabinets, arranged in one or more rows or columns or in anyother layout. In turn, ToR switches 306A, 306B, 306C (collectively“306”) can be connected to aggregation switch 304 typically viafiber-optic cabling. Aggregation switch 304 may be connected to otheraggregation switches or the rest of the network such as LAN, WAN, etc.(not shown).

FIG. 4 illustrates a schematic diagram of an example MU-MIMO network. Inexample wireless network 400, access point (AP) 402 can establishwireless links with one or more client devices such as client 1 (404-1),client 2 (404-2), client 3 (404-3), . . . and client N (404-n)(collectively “404”). AP 402 and clients 404 are equipped withtransceivers capable of transmitting and receiving RF signals. AP 402and clients 404 can transmit and receive radio frequency (RF) signals toand from each other to exchange data symbols. In some embodiments, thetransmitter and the receiver can use just a single antenna while in someother embodiments, they can use two or more antennas, thus achieving amultiple-input multiple-output (MIMO) link. When the MIMO link isbetween a single multi-antenna transmitter and a single multi-antennareceiver, the link is considered a single-user MIMO (SU-MIMO) link. MIMOis an integral part of many wireless communication standards such asInstitute of Electrical and Electronics Engineers (IEEE) 802.11n(Wi-Fi), IEEE 802.11ac (Wi-Fi), Evolved High-Speed Packet Access (HSPA+)(3G), Worldwide Interoperability for Microwave Access (WiMAX) (4G), andLong Term Evolution (LTE) (4G), and other standards currently indevelopment.

When MIMO is employed, multiple RF signals can be bundled up by way ofconstructive interference by performing a digital signal processingtechnique called “beam-forming,” which is also known as spatialfiltering. The process of beam-forming modulates the data symbols intoRF signals suited for transmission over a particular wireless channelbetween a transmitter and a receiver. The resulting “beam” is adirectional signal that is generated by the transmitter and directed atthe receiver. Since beam-forming can dynamically adapt to the conditionsand attributes of the particular communication channel, the fidelity ofthe communication link between the transmitter and the targeted receivercan be improved. Where there are two more or more receivers, eachreceiver's wireless channel with the transmitter can be different. Inother words, the transmitter can establish multiple wireless channels,each channel dedicated for each of the receivers. According to eachchannel's characteristics, the transmitter may also modulate multiplereceivers' data symbols together into one set of RF signals to be sentout concurrently. Then, even though the resulting RF signal containsdata symbols for multiple receivers, each targeted receiver can stilldecode its data symbols. Such technique of combining multiple RFsignals, as shown in FIG. 4, is called multi-user MIMO (MU-MIMO), wherethere may be two or more multi-antenna transmitters or two or moremulti-antenna receivers.

Specifically, when the MU-MIMO configuration occurs in the downlink(i.e., from one transmitter to two or more receivers), it is considereda point-to-multipoint link, or downlink multi-user multiple-input andmultiple-output (DL-MU-MIMO). Similar technique can be applied in theuplink when clients 404 are cognizant of the wireless channels betweenthem and AP 402, such that multiple clients' 404 data symbols aretransmitted concurrently on the uplink direction to AP 402. This isknown as a multipoint-to-point link, or uplink multi-user multiple-inputand multiple-output (UL-MU-MIMO).

FIG. 5 illustrates an example server rack coupled with a wirelesstransceiver. Server rack 502 may be an enclosure for mounting multipleequipment modules such as ToR switch 504 and blade servers 506-1, 506-2,. . . , 506-n (collectively “506”). Rack 502, ToR switch 504, andservers 506 may each correspond to rack 302A, ToR switch 306A, andservers 308A previously shown in FIG. 3. Servers 506 may be connected toswitch 504 via copper and/or fiber-optic cabling (not shown). Switch 504may in turn be connected to the rest of the network through anotherpiece of equipment such as an aggregation switch (not shown). Inaddition, ToR switch 504 may be connected to wireless transceiver 508.Wireless transceiver 508 may be capable of establishing MU-MIMO linkswith one or more other wireless transceivers, which can be connected totheir respective ToR switches. Thus, wireless transceiver 508 mayutilize its antenna(s) 510 to establish point-to-multipoint ormultipoint-to-point ToR flyways, as will be described in detail later.

FIG. 6 illustrates a schematic block diagram of an example configurationfor a data center augmented by wireless flyways. Similar to data centerconfiguration 300 shown in FIG. 3, data center configuration 600 mayconsist of racks 302, aggregation switch 304, ToR switches 306, andservers 308, which are interconnected to each other through wiredcabling such as copper and/or fiber. The wired network, then, canprovide the base network for servers 308. In other words, when oneserver or group of servers within data center 600 needs to exchange datawith another server or group of servers, it may do so via the basenetwork.

However, since the data transmission demands of servers 308 mayfluctuate over time, the data traffic may experience spikes that causecongestion in the base network. Such spikes may occur only infrequentlybut when they do occur, it can significantly affect the performance ofthe network and even bring down the entire network temporarily. However,it would be inefficient and costly to design the whole distributedcomputing system around the peak traffic because during non-peak hours(i.e., majority of the time), much of the equipment would be kept idle.This problem can become more pronounced as the distributed network growsin size. Thus, data centers can greatly benefit from a dynamicdeployment of flyways that can temporarily yet effectively alleviatedata congestion and improve overall performance of the network byproviding extra capacity to connect just a few ToR switches at a time.Flyways are wired or wireless links that are set up on demand orcommodity switches that interconnect random subsets of the ToR switches.Specifically, transceivers capable of MU-MIMO data transmissions canestablish wireless flyways to supplement the existing wired basenetwork. This wired/wireless hybrid approach allows the designers of thedata center to provision the network based on the average demandscenario instead of oversubscribing the network with excess equipment.

Accordingly, ToR switches 306 may be respectively coupled with wirelesstransceivers 602A, 602B, 602C (collectively “602”). The flyways mayutilize a particular frequency or a range of frequencies in the allottedfrequency spectrum. For example, transceiver 508 may implement MU-MIMOchannels in the 60 GHz frequency band to establish point-to-multipointor multipoint-to-point flyways. Due to the high attenuation at the 60GHz band, such links are typically short range (e.g., 1-10 meters)suitable for the typical distance between ToRs in a data center.Moreover, since a channel can be up to a few gigahertz wide, it cansupport high-bandwidth data transfer of 1 Gbps or higher. Accordingly,in addition to the wired base network, ToR switches 306 can wirelesslyroute data to and from servers 308 through transceivers 602. Forexample, when server 308A-3 suddenly finds itself in need oftransferring to servers 308B-1, 308C-2, and 308C-3 a large amount ofdata in excess of the data transmission capabilities of the basenetwork, rather than placing an undue stress on the wired base networkand thereby causing congestion, ToR switch 306A can use its transceiver602A to reroute the excess traffic wirelessly by establishing apoint-to-multipoint DL-MU-MIMO flyway with transceivers 602B and 602C.Conversely, as an example, when multiple servers 308A-3, 308C-1, 308C-2need to send data to server 308B-2, ToR switches 306A, 306C andtransceivers 602A, 602C can establish a multipoint-to-point wirelessflyway to ToR 306B and transceiver 602B to perform a UL-MU-MIMO wirelesstransmission.

In order to modulate signals according to a particular wireless channel,the properties and characteristics of the channel have to be known,typically ahead of time. These known channel properties are referred toas channel state information (CSI) and collectively they describe how aparticular signal may propagate from the transmitter to the receiver.These channel properties may include fading, scattering, decay, type offading distribution, average channel gain, line-of-sight component,spatial correlation, etc. Other attributes may include frequency, therelative speed at which the transmitter and/or receiver is moving, andwhether there are obstructions along or near the transmission path. Thevarious properties of the wireless channel can be measuredinstantaneously at discrete moments or statistically over an extendedperiod of time. The CSI may be obtained by exchanging training signalsbetween the transmitter and the receiver. In MU-MIMO, a set of CSI maybe obtained for each wireless channel, for example, between thetransmitter and each of the multiple receivers. Since the conditions andproperties of a wireless channel may fluctuate over time, depending onthe wireless environment, propagation medium, etc., training may have tobe performed frequently.

For example, in an exemplary data center environment with wirelesstransmitters operating at the 60 GHz frequency, which is relativelyhigh, the properties and attributes of the wireless channels may varyonly slightly. Moreover, because the locations of the ToR switches aretypically fixed and there are no moving objects near the ceiling toobstruct a line of sight, one may expect the channels to be nearlystatic. The relatively stable conditions of the data center maytranslate to no or infrequent needs for performing training to updatethe CSI for a given wireless communications channel.

Although data center 600 is illustrated in terms of ToR switches 602 andToR architecture, one of skill in the art will understand that disclosedembodiments can also apply to other types of data centers. For example,transceivers 602 may be attached to EoR or MoR switches to establishsimilar wireless links that complement the base network.

FIG. 7 illustrates a schematic diagram of an example data centerperforming a DL-MU-MIMO transmission among the ToR switches. In FIG. 7,exemplary data center 700 is shown in a top-down view, where serverracks are arranged in four rows with seven racks in each row. Serverrack 702 and server racks 704A, 704B, 704C, 704D, 704E (collectively“704”) are also shown as part of data center 700. This configuration,however, is merely exemplary and one of skill in the art will understandthat other configurations with fewer or more server racks are alsopossible. Racks 702, 704 may be similar to other racks previously shownin FIGS. 5 and 6. As such, racks 702, 704 may house therein switches,servers, and/or other modular equipment, and be interconnected through awired base network (not shown). Thus, in this description, racks 702,704 may also be referred to as ToR switches 702, 704. ToRs 702, 704 maybe also equipped with transceivers and antennas 706 and 708A, 708B,708C, 708C, 708D, 708E (collectively “708”) for establishing wirelesslinks and flyways. The transceivers may be, for example, 60 GHz MU-MIMOtransceivers connected to their respective ToR switches. One of skill inthe art, however, will realize that transceivers 706, 708 may transmitand receive signals at a radio frequency or band of radio frequenciesother than 60 GHz.

Antennas 706, 708 can be positioned and oriented in such a way tomaximize their abilities to transmit and receive radio signals. In oneexample, antenna 706 can be positioned in the general direction of othertransceivers such that it can achieve better communication performance.In some embodiments, the positions and directions of antennas 706, 708can be adjusted manually or automatically. Transceivers and antennas706, 708 may also periodically exchange relevant CSI among each othersuch that, should there be a need, a flyway may be created at a moment'snotice.

In this example, ToR 702 has data traffic queued up to be sent tomultiple other ToRs 704. ToR 702, ToRs 704, or another centralcontrolling device (not shown) may determine that transmitting the datatraffic to ToRs 704 may overwhelm the base network or otherwise congestthe network above a predetermined threshold. Thus, ToR 702, ToRs 704, orthe central controlling device may decide to send the traffic via apoint-to-multipoint flyway instead. In order to establish the wirelessflyway, ToR 706 may examine destination ToRs 704 and determine whetherany of them are within range of its 60 GHz transceiver 706. For example,ToR 706 may determine that ToRs 704A, 704B, 704C, 704D are well withinthe range that would guarantee the predetermined minimum thresholdsignal strength for the wireless channels between ToR 706 and each ofToRs 704A, 704B, 704C, 704D, but rule out ToR 704E because it is too faraway from ToR 702. For those destination ToRs that are not in range, ToR702 can rely on the base network to route the data traffic.

On the other hand, for those destination ToRs that are within range, ToR702, ToRs 704, or the central controlling device may consider whetherthe CSI between ToR 702 and each of the destinations is up to date(i.e., last updated within the past X seconds, where X is a configurableparameter; for example, X=50 ms). In a data center that is more or lessstatic and does not involve too many moving components, the updatefrequency can be relatively low while the more dynamic data center withfrequent changes to its configuration and layout may call for morefrequent CSI updates. For those destination ToRs with outdated CSI, ToR702 can route their data traffic by the base network or let the datatraffic remain in the transmission queue until the appropriate CSI isupdated in the next scheduled training session. Alternatively, ToR 702may initiate another training session immediately to update thenecessary CSI for the outdated ToRs.

Now, for those destination ToRs with up-to-date CSI, ToR 702,transceiver 706, or the central controlling device may select the firstY destinations (where Y is a configurable parameter and Y>1) in thequeue and calculate, based on their respective CSI, a beam-formingcoefficient to be applied to data symbols for generating the RF signals.Y may depend on the number of antennas available in each multi-antennatransceiver. For example, Y may be 4. If the calculation shows that theresulting beam-forming is suboptimal (i.e. fails to effectively use theavailable wireless spectrum and spatial diversity), ToR 702, transceiver706, or the central controlling device may select other destination ToRsthat are further down in the queue. In turn, ToR 702 may route the datatraffic that was destined for the unfit candidate ToRs through the basenetwork instead.

Once the Y candidate destination ToRs are selected, vetted, andfinalized, transceiver 706 can transmit an RF signal via MU-MIMO toselected Y destination ToRs 704A, 704B, 704C, 704D. The RF signal can begenerated by precoding and/or spatial multiplexing according to the CSIcollected from destination ToRs. Each destination ToR may receive the RFsignal and decode from it the data symbols intended for the respectivedestination ToR.

The wireless flyways may also interoperate with the rest of datacenter's 700 network (i.e., base network) to route and forward datapackets. Such operations may occur via the control plane at Layer 3.Since an MU-MIMO flyway is not a symmetric link (i.e., it is eitheruplink or downlink), when transceiver 706 transmits to multipledestination ToRs 708A, 708B, 708C, 708D, the link is unidirectional.This is because the destination ToRs can typically only operate inreceive mode once the link is established and cannot transmit back toToR 706. However, most network communication protocols require that theend hosts (e.g., servers) exchange control packets, such as anacknowledgement (ACK) packet to signal successful receipt of data. Thus,destination ToRs 708A, 708B, 708C, 708D require reverse channels (alsocalled “reverse communications paths”) that lead back to ToR 702 suchthat the destination ToRs can route any control packets transmitted byany destination hosts back to the source host via source ToR 706.

In some embodiments, destination ToRs 708A, 708B, 708C, 708D may firstterminate the established point-to-multipoint MIMO link and thenestablish another link in the opposite direction to transmit data suchas the control packets back to ToR 702. However, this approach can becostly because of the overhead involved in establishing and terminatingmultiple links in succession. Thus, in some other embodiments, ToRs mayomit setting up such reverse channels during a point-to-multipoint MIMOtransmission such that ToR 702 can simply “blast” away at high bandwidthon the 60 GHz link in one direction without needing to stop andre-establish links periodically. Instead, ToRs 702, 704 may utilize theexisting wired base network for reverse channel transmissions. That is,in addition to having a wireless interface, each ToR may also haveaccess to the wired network such as Ethernet via copper and/or fibercabling. Since the wired links in the base network are bi-directional,when a given destination ToR is due for sending control packets (or insome cases a small amount of data packets) back to ToR 702, thedestination ToR may do so by sending the packets through data center's700 wired network, which would then route the packets to ToR 702. Sincecontrol packets are typically small or even negligible in size, theywould not create any congestion in the wired network and they can berouted to ToR 702 expeditiously. By employing a bi-directional channelemulation scheme such as UniDirectional Link Routing (UDLR), from theviewpoint of the network's Layer-3 routing mechanism, it is possible tomake the underlying wireless paths appear as though they have a reversepath through the wireless link when in fact the reverse traffic wouldactually traverse the wired links. Thus, ToRs 702, 704 may achievepoint-to-multipoint MIMO connectivity without the need for terminatingthe wireless link currently in session. Diverting the reverse trafficcan be accomplished by tunneling any packets for a flow carried on theMU-MIMO channel that arrive at ToR 704. That is, a packet can beencapsulated with ToR 704 as the source and ToR 702 as the destination,and the route from ToR 704 to ToR 702 can be set to a next hop router onthe wired network. Alternatively, data center 700 can insert a “sourceroute” in a packet in the reverse traffic where the source routeexplicitly points the packet to a next hop on the wired network. Otherrouting methods may be possible. For example, data center 700 can use asoftware defined network (SDN) controller to directly program the dataplane of the wired switches to ensure that packets for a flow carried onthe MU-MIMO channel arriving at ToR 704 whose next hop would normally beToR 702 go through a series of switches on the wired network.

As a corollary, as Layer 3 becomes aware of the potentialpoint-to-multipoint data path patterns that can offer additional highbandwidths, Layer 3 may establish point-to-multipoint MIMO flyways thatnot only do not interfere with each other, but also optimize totaloverall network capacity. In addition, since the wireless flyways thathave been set up now appear to Layer 3 as symmetric (i.e., having areverse channel), the high-bandwidth forward paths can continue tooperate without interference until the data flow is complete. When thepoint-to-multipoint flow pattern changes afterwards, the control planeat Layer 3 may only need to modify the forwarding information bases(FIBs) accordingly. The aforementioned concurrent point-to-multipointMIMO data flows can improve the overall capacity of the network overdynamically occurring traffic patterns. Furthermore, by utilizing thewireless flyways, the network topology becomes more fluid and dynamiccompared to the fixed network topology of the conventional wired datacenters. Although this hybrid wired/wireless approach is described interms of a point-to-multipoint MU-MIMO transmission as illustrated inFIG. 7, it may also apply to multipoint-to-point MU-MIMO transmissionsas shown in FIG. 8.

FIG. 8 illustrates a schematic diagram of an example data centerperforming a UL-MU-MIMO transmission among the ToR switches. As withFIG. 7, exemplary data center 800 is shown in a top-down view, whereserver racks are arranged in rows and columns. Server rack 802 (oralternatively ToR 802) and server racks 804A, 804B, 804C, 804D, 804E (oralternatively ToRs 804A, 804B, 804C, 804D, 804E) (collectively “804”)are also shown as part of data center 800. This configuration, however,is merely exemplary and one of skill in the art will understand thatother configurations with fewer or more server racks and switches arealso possible. Racks 802, 804 may be similar to other racks previouslyshown in FIGS. 5-7. As such, racks 802, 804 may house therein switches,servers, and/or other modular equipment, interconnected through a wiredbase network (not shown). ToRs 802, 804 may be equipped withtransceivers and antennas 806 and 808A, 808B, 808C, 808C, 808D, 808E(collectively “808”) for establishing wireless links and flyways. Thetransceivers may be, for example, 60 GHz MU-MIMO transceivers connectedto their respective ToR switches. One of skill in the art, however, willrealize that transceivers 806, 808 may transmit and receive radiosignals at different radio frequency or band of radio frequencies.

Antennas 806, 808 can be positioned and adjusted manually orautomatically to achieve better communication performance. Transceiversand antennas 806, 808 may also periodically exchange relevant CSI amongeach other. In this example, multiple ToRs 804 wish to send data to oneToR 802. Destination ToR 802, for example, can be a popular data sinksuch as a redundant backup server or a job history logger. However, ToRs804, ToR 802, or a central controlling device (not shown) may determinethat transmitting the data traffic from ToRs 804 to ToR 802 through thewired base network may be too burdensome for the base network. Thus,ToRs 804, ToR 802, or the central controlling device may decide thatsending the traffic via a wireless flyway would be a better choiceinstead. Such a flyway can be a multipoint-to-point flyway employingUL-MU-MIMO.

In order to establish the multipoint-to-point flyway, ToRs 804, ToR 802,or the central controlling device may determine whether source ToRs 804are within range of destination ToR 802. For those source ToRs that arenot in range, those ToRs may choose to rely on the base network to routetheir traffic. For example, in data center 800, it may be determinedthat ToRs 804C and 804E are actually too far away from the intendeddestination, ToR 802. Thus, transceivers 808C, 808E may not be able toestablish a reliable wireless channel (e.g., having at least the minimumthreshold signal strength) with transceiver 806. In such a case, ToRs804C, 804E may simply use the underlying base network to transmit datato ToR 802.

For those source ToRs that are within range, however, ToRs 804, ToR 802,or the central controlling device may consider whether the CSI betweenToR 802 and each of the sources is up to date (i.e., last updated withinthe past X seconds, where X is a configurable parameter; for example,X=50 ms). The required update frequency may depend on the individualcircumstances and characteristics of a given data center. Those sourceToRs with outdated CSI may choose to route their data to destination ToR802 by the base network. Alternatively, they can wait until the nexttraining session to update the latest CSI and then send the data trafficvia a flyway.

Once the CSI has been updated, ToRs 804, ToR 802, or the centralcontrolling device may select two or more flyway candidate source ToRsbased on their CSI. This can be done by calculating which combination ofsource ToRs is better at mixing their data symbols to form a UL-MU-MIMOsignal. In other words, the system can try to determine whichcombination of source ToRs may result in a more desirable beam-formingcoefficient. Accordingly, ToRs 804, ToR 802, or the central controllingdevice may select Y number of source ToRs as candidates, where Y is aninteger greater than 1. The corresponding beam-forming coefficient maythen be applied to each of the Y source ToRs' data symbols and eachsource ToR may generate an RF signal to transmit. These RF signals maythen be combined to form an MU-MIMO wireless link.

Destination ToR 802 can receive the RF signals sent concurrently in theair by the Y source ToRs. Since the data symbols of the respectivesource ToRs are beam-formed to compose a MU-MIMO signal, destination ToR802 can separate and decode individual groups of data symbols from eachsource ToR. Thus, many principles and concepts illustrated in FIG. 8 aresimilar to those of FIG. 7 but are applied in the reverse direction. Asone of ordinary skill in the art will readily recognize, the examplesand technologies provided above are simply for clarity and explanationpurposes, and can include many additional concepts and variations.

Having disclosed some basic system components and concepts, thedisclosure now turns to the exemplary method embodiments shown in FIGS.9-11. For the sake of clarity, the methods are described in terms ofsystem 110, as shown in FIG. 1, configured to practice the method.Alternatively, the methods may be practiced by system 200 of FIG. 2A orsystem 250 of FIG. 2B. The steps outlined herein are exemplary and canbe implemented in any combination thereof, including combinations thatexclude, add, or modify certain steps. The steps may be practiced in theorder shown in FIGS. 9-11 or in any other order. Additionally, themethods illustrated in FIGS. 9-11 may also apply to types of switchesother than ToR switches, such as EoR and MoR switches and the like.

FIG. 9 illustrates an example method embodiment of performing aDL-MU-MIMO transmission. System 110 may identify, at a source ToR (alsoknown as top-of-rack switch) connected to a group of ToRs via a basenetwork, a plurality of destination ToRs from the group of ToRs to senddata traffic to (902). Each of the ToRs may be placed in a rack andconnect to other servers, storage, equipment, etc. in the rack. ToRs maybe also connected to an aggregation switch as part of the base network.The base network can be a wired network based on copper and/orfiber-optic cables. System 110 may determine whether each destinationToR of the plurality of destination ToRs is suitable for receiving datatransmission via a point-to-multipoint wireless flyway, to yield two ormore flyway candidate ToRs (904). This may include determining whetherthe each destination ToR is within a threshold range of a transceiverassociated with the source ToR. When a particular destination ToR isdetermined to be out of the threshold range of the transceiver, thesource ToR can route the data traffic to the particular destination ToRvia the base network.

Determining suitability may also include determining whether respectivechannel state information (CSI) associated with the each destination ToRhas been received at the source ToR within a predetermined time in apast. In other words, the determination may be made as to whether theCSI associated with a particular destination ToR is up to date. When CSIassociated with a particular destination ToR switch of the plurality ofdestination ToR switches has not been received at the source ToR switchwithin the predetermined time in the past, source ToR may route the datatraffic to the particular destination ToR switch via the base network.Alternatively, the source ToR may delay transmission of the traffic datato the particular destination ToR switch until updated CSI is receivedfrom the particular destination ToR switch. Thus, the two or more flywaycandidate ToRs may be selected among the plurality of destination ToRsaccording to one or more criteria discussed above. The number ToRs inthose two or more flyway candidate ToRs can be a tunable parameter thatdepends on a number of antennas available for establishing thepoint-to-multipoint wireless flyway. For example, the number antennasavailable for the source ToR and the number of antennas available forthe destination ToRs may influence how many flyway candidate ToRs arechosen.

System 110 may establish the point-to-multipoint wireless flyway betweenthe source ToR and the two or more flyway candidate ToRs (906). This canbe accomplished by calculating a beam-forming coefficient based onrespective CSI associated with each of the two or more flyway candidateToR switches. The calculated beam-forming coefficient may then beapplied to data symbols for generating an RF signal. However, when thebeam-forming coefficient calculated based on the respective CSI fails tomeet a threshold condition, a different group of destination ToRs may beselected out of the plurality of destination ToRs as flyway candidateToRs. The threshold condition may be related to the signal strength,reliability, throughput, etc. of the resulting beam.

System 110 may transmit the data traffic from the source ToR to the eachflyway candidate ToR of the two or more flyway candidate ToRs via thepoint-to-multipoint wireless flyway (908). This may be accomplished bytransmitting an RF signal from a transceiver associated with the sourceToR switch to a respective transceiver associated with the each flywaycandidate ToR switch via a downlink multi-user multiple-input andmultiple-output (DL-MU-MIMO) signal. The source ToR may receive acontrol packet from the each flyway candidate ToR switch via the basenetwork.

FIG. 10 illustrates an example method embodiment of performing aUL-MU-MIMO transmission. System 110 may identify a plurality of sourceToRs from a group of ToRs interconnected to each other via a wired basenetwork, each source ToR of the plurality of source ToRs havingrespective data to transmit to a destination ToR of the group of TORs(1002). Based on respective CSI received from the each source ToR,system 110 may select two or more flyway candidate ToRs from theplurality of source ToRs, the two or more flyway candidate ToRs having abeam-forming coefficient that satisfies a threshold value (1004). Theselection of the two or more flyway candidate ToRs may be also performedby determining whether the each source ToR is within a threshold rangeof a transceiver associated with the destination ToR.

System 110 may establish the multipoint-to-point wireless flyway betweenthe two or more flyway candidate ToRs and the destination ToR (1006).System 110 may apply the beam-forming coefficient to respective datasymbols of the each source ToR, where the each source ToR is configuredto generate an RF signal to transmit. System 110 may receive, at thedestination ToR, respective data symbols from the two or more flywaycandidate ToRs via the multipoint-to-point wireless flyway (1008). Thedestination ToR may concurrently receive RF signals transmitted from thetwo or more flyway candidate ToRs. The RF signals may have beenbeam-formed to form an uplink multi-user multiple-input andmultiple-output (UL-MU-MIMO) signal.

FIG. 11 illustrates an example method embodiment of routing reversetraffic from a destination ToR switch to a source ToR switch. Asdiscussed above, although MU-MIMO links are capable of providing ahigh-volume traffic pathway that bypasses the wired network, thistraffic is typically unidirectional (i.e., it travels from the sourceToR to the destination ToR but not the other way around). However, somepackets may need to go back over a reverse path from the destination ToRto the source ToR as well in order to keep the link up and carryresponse traffic such as TCP ACKs. In order to ensure that the reversetraffic (i.e., traffic that would normally flow in the oppositedirection of a full-duplex link) does not require the establishment of aseparate MI-MIMO radio channel and can be carried over the wirednetwork, system 110 can employ a UDLR routing.

In particular, system 110 can use UDLR to avoid having to establish areverse channel over the wireless flyway. Thus, system 110 can establisha multi-user MIMO wireless flyway from a source ToR to a destination ToR(1102). Then, system 110 can create a routing adjacency between thesource ToR and the destination ToR (1104). System 110 can mark therouting adjacency at the destination ToR switch as UDLR (1106), with thenext hop for packets about to be forwarded over the reverse directionfrom the destination ToR to the source ToR to instead go over the wirednetwork. The routing adjacency may be created on both the source ToR andthe destination ToR to represent the link. Once such adjacency iscreated, the source-to-destination traffic can flow over the MU-MIMOchannel while the destination-to-source traffic (i.e., reverse traffic)may be diverted to the wired network such that when packets arrive atthe destination ToR for a destination for which the best next routinghop is the source ToR, the packets can go over the wired network. Thus,system 110 can transmit, from the destination ToR, a packet with a nexthop to the source ToR via a wired network (1108).

System 110 can transmit the reverse-traffic packet from the destinationToR to the source ToR via the wired network by encapsulating the packetwith the destination ToR as the “source” and the source ToR as the“destination, and tunneling the packet by setting the route to thesource ToR to a next-hop router on the wired network. Alternatively,system 110 may also insert a “source route” in the reverse-trafficpacket explicitly pointing the packet to a next hop on the wirednetwork. System 110 may also use an SDN controller to directly programthe data plane of the wired switches to ensure that the packets arrivingat the destination ToR whose next hop would normally be the source ToRgo through a series of switches on the wired network instead. When themulti-user MIMO wireless flyway is terminated, system 110 may alsoremove the routing adjacency created between the source ToR and thedestination ToR (1110).

For clarity of explanation, in some instances the present technology maybe presented as including individual functional blocks includingfunctional blocks comprising devices, device components, steps orroutines in a method embodied in software, or combinations of hardwareand software.

In some embodiments the computer-readable storage devices, mediums, andmemories can include a cable or wireless signal containing a bit streamand the like. However, when mentioned, computer-readable storage mediaor devices expressly exclude transitory media such as energy, carriersignals, electromagnetic waves, and signals per se.

Methods according to the above-described examples can be implementedusing computer-executable instructions that are stored or otherwiseavailable from computer readable media or devices. Such instructions caninclude, for example, instructions and data which cause or otherwiseconfigure a general purpose computer, special purpose computer, orspecial purpose processing device to perform a certain operation orgroup of operations. Portions of computer resources used can beaccessible over a network. The computer executable instructions may be,for example, binaries, intermediate format instructions such as assemblylanguage, firmware, or source code. Examples of computer-readable mediathat may be used to store instructions, information used, and/orinformation created during methods according to described examplesinclude magnetic or optical disks, flash memory, Universal Serial Bus(USB) devices provided with non-volatile memory, networked storagedevices, and so on.

Devices implementing methods according to these disclosures can comprisehardware, firmware and/or software, and can take any of a variety ofform factors. Typical examples of such form factors include laptops,smart phones, small form factor personal computers, personal digitalassistants, rackmount devices, standalone devices, and so on.Functionality described herein also can be embodied in peripherals oradd-in cards. Such functionality can also be implemented on a circuitboard among different chips or different processes executing in a singledevice, by way of further example.

The instructions, media for conveying such instructions, computingresources for executing them, and other structures for supporting suchcomputing resources are means for providing the functions described inthese disclosures.

Although a variety of examples and other information was used to explainaspects within the scope of the appended claims, no limitation of theclaims should be implied based on particular features or arrangements insuch examples, as one of ordinary skill would be able to use theseexamples to derive a wide variety of implementations. Further andalthough some subject matter may have been described in languagespecific to examples of structural features and/or method steps, it isto be understood that the subject matter defined in the appended claimsis not necessarily limited to these described features or acts. Forexample, such functionality can be distributed differently or performedin components other than those identified herein. Rather, the describedfeatures and steps are disclosed as examples of components of systemsand methods within the scope of the appended claims. Thus, the claimsare not intended to be limited to the aspects shown herein, but are tobe accorded the full scope consistent with the language claims, whereinreference to an element in the singular is not intended to mean “one andonly one” unless specifically so stated, but rather “one or more.”

A phrase such as an “aspect” does not imply that such aspect isessential to the subject technology or that such aspect applies to allconfigurations of the subject technology. A disclosure relating to anaspect may apply to all configurations, or one or more configurations. Aphrase such as an aspect may refer to one or more aspects and viceversa. A phrase such as a “configuration” does not imply that suchconfiguration is essential to the subject technology or that suchconfiguration applies to all configurations of the subject technology. Adisclosure relating to a configuration may apply to all configurations,or one or more configurations. A phrase such as a configuration mayrefer to one or more configurations and vice versa.

The word “exemplary” is used herein to mean “serving as an example orillustration.” Any aspect or design described herein as “exemplary” isnot necessarily to be construed as preferred or advantageous over otheraspects or designs. Moreover, claim language reciting “at least one of”a set indicates that one member of the set or multiple members of theset satisfy the claim.

We claim:
 1. A method comprising: identifying, at a source top-of-rack(ToR) switch connected to a group of ToR switches via a base network, aplurality of destination ToR switches from the group of ToR switches tosend data traffic to; determining whether each destination ToR switch ofthe plurality of destination ToR switches is suitable for receiving datatransmission via a point-to-multipoint wireless flyway, to yield two ormore flyway candidate ToR switches; establishing the point-to-multipointwireless flyway between the source ToR switch and the two or more flywaycandidate ToR switches; and transmitting the data traffic from thesource ToR switch to the each flyway candidate ToR switch of the two ormore flyway candidate ToR switches via the point-to-multipoint wirelessflyway.
 2. The method of claim 1, wherein the base network is a wirednetwork.
 3. The method of claim 1, wherein the each destination ToRswitch is determined to be part of the two or more flyway candidate ToRswitches when the each destination ToR switch is within a thresholdrange of a transceiver associated with the source ToR switch.
 4. Themethod of claim 3, further comprising: when a particular destination ToRswitch of the plurality of destination ToR switches is determined to beout of the threshold range of the transceiver, routing the data trafficfrom the source ToR switch to the particular destination ToR switch viathe base network.
 5. The method of claim 1, wherein the each destinationToR switch is determined to be part of the two or more flyway candidateToR switches when respective channel state information (CSI) associatedwith the each destination ToR switch has been received at the source ToRswitch within a predetermined time in a past.
 6. The method of claim 5,further comprising: when CSI associated with a particular destinationToR switch of the plurality of destination ToR switches has not beenreceived at the source ToR switch within the predetermined time in thepast, routing the data traffic to the particular destination ToR switchvia the base network.
 7. The method of claim 5, further comprising: whenCSI associated with a particular destination ToR switch of the pluralityof destination ToR switches has not been received at the source ToRswitch within the predetermined time in the past, delaying transmissionof the traffic data from the source ToR switch to the particulardestination ToR switch until updated CSI is received from the particulardestination ToR switch.
 8. The method of claim 1, wherein a number ofthe two or more flyway candidate ToR switches is a tunable parameterthat depends on a number of antennas available for establishing thepoint-to-multipoint wireless flyway.
 9. The method of claim 1, whereinestablishing the point-to-multipoint wireless flyway comprisescalculating a beam-forming coefficient based on respective CSIassociated with each of the two or more flyway candidate ToR switches.10. The method of claim 9, wherein the beam-forming coefficient is to beapplied to data symbols for generating a radio frequency (RF) signal.11. The method of claim 9, wherein, when the beam-forming coefficientcalculated based on the respective CSI fails to meet a thresholdcondition, selecting a different group of destination ToR switches ofthe plurality of destination ToR switches to be the two or more flywaycandidate ToR switches.
 12. The method of claim 1, wherein transmittingthe data traffic comprises transmitting an RF signal from a transceiverassociated with the source ToR switch to a respective transceiverassociated with the each flyway candidate ToR switch via a downlinkmulti-user multiple-input and multiple-output (DL-MU-MIMO) signal. 13.The method of claim 1, further comprising: receiving, at the source ToRswitch, a control packet from the each flyway candidate ToR switch viathe base network.
 14. The method of claim 1, further comprising:creating a routing adjacency between the source ToR switch and the oneof the two or more flyway candidate ToR switches; marking, at the one ofthe two or more flyway candidate ToR switches, the routing adjacency asa unidirectional link routing (UDLR); receiving, at the source ToRswitch and via the base network, a packet from the one of the two ormore flyway candidate ToR switches, the packet having a next hop to thesource ToR; and when the point-to-multipoint wireless flyway isterminated, removing the routing adjacency.
 15. A method comprising:identifying a plurality of source ToR switches from a group of ToRswitches interconnected to each other via a wired base network, eachsource ToR switch of the plurality of source ToR switches havingrespective data to transmit to a destination ToR switch of the group ofToR switches; based on respective CSI received from the each source ToRswitch, selecting two or more flyway candidate ToR switches from theplurality of source ToR switches, the two or more flyway candidate ToRswitches having a beam-forming coefficient that satisfies a thresholdvalue; establishing the multipoint-to-point wireless flyway between thetwo or more flyway candidate ToR switches and the destination ToRswitch; and receiving, at the destination ToR switch, respective datasymbols from the two or more flyway candidate ToR switches via themultipoint-to-point wireless flyway.
 16. The method of claim 15,selecting the two or more flyway candidate ToR switches furthercomprises determining whether the each source ToR switch is within athreshold range of a transceiver associated with the destination ToRswitch.
 17. The method of claim 15, further comprising: applying thebeam-forming coefficient to respective data symbols of the each sourceToR switch, wherein the each source ToR switch is configured to generatean RF signal to transmit.
 18. The method of claim 15, wherein receivingthe respective data symbols comprises concurrently receiving RF signalstransmitted from the two or more flyway candidate ToR switches.
 19. Themethod of claim 16, wherein the RF signals are beam-formed to compose anuplink multi-user multiple-input and multiple-output (UL-MU-MIMO)signal.
 20. An apparatus comprising: a source top-of-rack (ToR) switchconnected to a group of ToR switches via a base network; a processor;and a computer-readable storage medium storing instructions which, whenexecuted by the processor, cause the processor to perform operationscomprising: identifying, at the source ToR switch, a plurality ofdestination ToR switches from the group of ToR switches to send datatraffic to; determining whether each destination ToR switch of theplurality of destination ToR switches is suitable for receiving datatransmission via a point-to-multipoint wireless flyway, to yield two ormore flyway candidate ToR switches; establishing the point-to-multipointwireless flyway between the source ToR switch and the two or more flywaycandidate ToR switches; and transmitting the data traffic from thesource ToR switch to the each flyway candidate ToR switch of the two ormore flyway candidate ToR switches via the point-to-multipoint wirelessflyway.